Track CustomContainerImages and prevent minor updates w/o images updates#1569
Conversation
|
letting CI run before I mark as draft. This is missing code to allow the user to override the restriction if they want to, essentially forcing the change regardless of the warning if they wish |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3226a8e13a484ce28308f2ef50e3fec7 ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 17m 27s |
dprince
force-pushed
the
tracked_custom_images
branch
from
a760b91 to
d5224a3
Compare
dprince
force-pushed
the
tracked_custom_images
branch
from
d5224a3 to
a4cbd4c
Compare
dprince
force-pushed
the
tracked_custom_images
branch
2 times, most recently
from
dc96ab3 to
7730fca
Compare
dprince
force-pushed
the
tracked_custom_images
branch
from
7730fca to
2a416d7
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/332b4e1f7f764474aa85e40e1132f0ba ❌ openstack-k8s-operators-content-provider POST_FAILURE in 3h 24m 09s |
for safety mechanism I think that makes sense. We'd have to document that behavior in minor updates doc. |
…hanges Add tracking and validation to ensure CustomContainerImages are updated when changing targetVersion during minor updates: - Add TrackedCustomImages field to OpenStackVersionStatus to track CustomContainerImages used for each version - Implement validation webhook logic to prevent targetVersion updates when CustomContainerImages remain unchanged from previous version - Add helper functions for deep comparison of CustomContainerImages - Update controller to automatically track CustomContainerImages for each processed target version - Generate updated CRD manifests with new tracking field This prevents inconsistent deployments where users update the target version but forget to update associated custom container images, ensuring proper version tracking and validation during minor updates. Users can skip all validations related to this commit by setting the core.openstack.org/skip-custom-images-validation metadata annotations. Co-Authored-By: Claude <noreply@anthropic.com> Jira: OSPRH-19183
dprince
force-pushed
the
tracked_custom_images
branch
from
2a416d7 to
e97685d
Compare
| instance.Status.DeployedVersion = &instance.Spec.TargetVersion | ||
|
|
||
| // Remove skip-custom-images-validation annotation after minor update completion | ||
| if instance.Annotations != nil { |
There was a problem hiding this comment.
Won't this remove the annotation on any reconcile when the ctlplane is ready, not just when a minor update finished? Like I do the minor update, it finishes, after that I want to re-add the condition. Maybe we should do it above after the dataplane got the minor update?
instance.Status.Conditions.MarkTrue(
corev1beta1.OpenStackVersionMinorUpdateDataplane,
corev1beta1.OpenStackVersionMinorUpdateReadyMessage)
There was a problem hiding this comment.
it would only remove the annotation if targetVersion != availableVersion. My intention with the annotation is that you would set it only when enabling a minor update. If we move it above after the dataplane gets update it is possible someone could set the annotation at anytime and it would persist until the next minor update and then it would skip the check blindly.
There was a problem hiding this comment.
right, thanks for clarification
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dprince, stuggi The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
1 similar comment
|
/retest |
|
/retest |
1 similar comment
|
/retest |
openshift-merge-bot
Bot
merged commit 70a918b
into
openstack-k8s-operators:main
…g minor update Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. During the update flow, `set_openstack_containers` updates the operator CSV env vars with the new delorean tag, but nothing updates the `OpenStackVersion` CR's `customContainerImages` -- they still carry the tag from the initial greenfield deployment. When `update_services` later patches `targetVersion`, the webhook rejects the request because the custom images haven't changed. Add a step after `set_openstack_containers` that reads the current `customContainerImages` from the `OpenStackVersion` CR, replaces every image tag with the new delorean md5 hash, and patches the CR. This ensures the custom images reflect the new version before `targetVersion` is changed, satisfying the webhook and guaranteeing the update uses the correct container images. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>
Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. In CI, the greenfield deployment and the simulated version bump share the same delorean hash, so `customContainerImages` are legitimately identical across versions -- there are no new container builds to reference. Retagging is a no-op. Add the operator-provided `skip-custom-images-validation` annotation before patching `targetVersion` in both update scripts. This annotation is the intended escape hatch for automation where the images are intentionally unchanged (see `ValidateUpdate` in `openstackversion_webhook.go`). The intermediate `openstackversionpatch.yaml` file is no longer generated; the patch is passed inline instead. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>
Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. In CI, the greenfield deployment and the simulated version bump share the same delorean hash, so `customContainerImages` are legitimately identical across versions. Clear `customContainerImages` (set to `null`) in the same merge patch that updates `targetVersion`. With no custom images present the webhook's `hasAnyCustomImage` check returns false and the validation is bypassed. The operator falls back to its CSV defaults, which `set_openstack_containers` has already updated to the correct tag. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>
…on` patch Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. In CI the same delorean hash is used for both the greenfield deployment and the simulated version bump, so `customContainerImages` are legitimately identical. Simply clearing them is not safe because the operator falls back to CSV defaults which may point to an inaccessible upstream registry (`quay.io`), while the custom images point to the internal mirror (`images.paas.redhat.com`). Work around this by saving `customContainerImages`, nulling them in the same merge patch that sets `targetVersion` (so the webhook's `hasAnyCustomImage` check returns false and skips validation), then immediately restoring them before the operator reconciles. This preserves the correct internal-mirror image references while bypassing the webhook. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>
3 participants
Add tracking and validation to ensure CustomContainerImages are updated when changing targetVersion during minor updates:
This prevents inconsistent deployments where users update the target version but forget to update associated custom container images, ensuring proper version tracking and validation during minor updates.
Co-Authored-By: Claude noreply@anthropic.com
Jira: OSPRH-19183